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Abstract 

We study matter effects wfiicfi arise in tfie muon neutrino osciliation and survivai prob- 
abiiities relevant to atmospheric neutrino and very long baseline (> 4000 Km) beam experi- 
ments. The inter-relations between the three probabilities P^e, P/xt and P^^ are examined. It 
is shown that large and observable sensitivity to the neutrino mass hierarchy can be present 
in P^^ and P^r- We emphasize that at baselines > 7000 Km, matter effects in P^,- are impor- 
tant under certain conditions and can be large. The muon survival rates in experiments with 
very long baselines thus depend on matter effects in both P^,- and P^e- We also indicate 
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where these effects provide sensitivity to ^13 and identify ranges of energies and baselines 
where this sensitivity is maximum. The effect of parameter degeneracies in the three prob- 
abiUties at these baseUnes and energies is studied in detail and large parts of the parameter 
space are identified which are free from these degeneracies. In the second part of the paper, 
we focus on using the matter effects studied in the first part as a means of determining the 
mass hierarchy via atmospheric neutrinos. Realistic event rate calculations are performed for 
a charge discriminating 100 kT iron calorimeter which demonstrate the possibility of realising 
this very important goal in neutrino physics. It is shown that for atmospheric neutrinos, a 
careful selection of energy and baseline ranges is necessary in order to obtain a statistically 
significant signal, and that the effects are largest in bins where matter effects in both P^e 
and P^T- combine constructively. Under these conditions, up to a 4(T signal for matter effects 
is possible (for A31 > 0) within a timescale appreciably shorter than the one anticipated for 
neutrino factories. 



Keywords : Neutrino Oscillations, Atmospheric neutrinos. Long baselines. Matter effects. 



1 Introduction 

It is fair to say that over the last few years there has been a qualitative shift in the nature of the 
goals to be pursued in neutrino physics. This has been the result of steadily accumulating evidence 
in favour of non-zero neutrino mass and flavour oscillations. Results from atmospheric neutrino 
experiments [1, 2, 3, 4, 5, 6, 7], corroborated by the accelerator beam based KEK to Kamioka 
(K2K) experiment [8] have provided firm evidence for i/^ z/^ oscillations with maximal (or 
almost maximal) mixing. The solar neutrino results [9, 10, 11, 12, 13, 14, 15, 16], when combined 
with the results of the reactor based KamLAND experiment [17], have established the LMA- 
MSW solution [18] as the most favoured explanation for the solar neutrino deficit. For recent 
global analyses of solar, reactor, accelerator and atmospheric, data, see [19, 20, 21, 22]. In Table 
1, we summarize the best-fit values and 3a intervals of allowed values of important oscillation 
parameters gleaned from experiments so far obtained from global three fiavour neutrino analysis 
[19, 21]. 

The shift has been from a search for understanding the particle physics and/or the astrophysics 
driving the solar and atmospheric neutrino deficits to one where we seek to make increasingly 
precise measurements of neutrino mass and mixing matrix parameters. Experiments planned to 
yield results over the next ten to fifteen years thus refiect this change of emphasis. A significant 
number of the planned projects arc long baseline^ endeavours using cither (a) a conventional 
proton beam colliding with a target to produce pions which then decay to give muon neutrinos, 
or (b) superbeams, which are essentially technologically upgraded versions of present conventional 

^By "long baseline" we actually mean the L/E range of about 50-500 Km/GeV. For accelerator experiments, 
this translates to baselines conventionally termed "long" , but for the lower reactor neutrino energies, the baselines 
are actually 1-2 Km. 
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Parameter 


Best-fit value 


3(7 allowed range 


A2i[10-5eV^] 
A3i[10-3eV^] 
sin^ di2 
sin^ 6I23 
sin^ ^13 


8.3 

2.2 

0.30 

0.50 

0.00 


7.2 - 9.1 
1.4-3.3 
0.23 - 0.38 
0.34 - 0.68 
< 0.047 



Table 1: Best- fit values and Sa intervals for three flavour neutrino oscillation parameters from global data 
including solar, atmospheric, reactor (KamLAND and CHOOZ) and accelerator (K2K) experiments [19, 21]. Here 



beams, or, finally, (c) reactor sources with both near and far detectors for reduced systematic 
errors. 

To begin with, we enumerate and briefly describe the planned projects in the three categories 
above which are expected to give results over the next decade or decade and a half. For a recent 
detailed study of their capabilities we refer the reader to [23] and references therein. 

• Conventional beam experiments will have as their primary goal the improvement in the 
precision of the atmospheric oscillation parameters, A31 and sin^ 2^23- The Main Injector 
Neutrino Oscillation Search (MINOS) project [24], located in the US, with a baseline 
from Fermilab to Soudan mine in the US state of Minnesota (735 Km), will utilize a 5.4 kT 
magnetized iron calorimeter and initially an < E,^ >~ 3 GeV to primarily measure muon 
survival events. By measuring the absolute event rate and the energy distribution for muons 
produced via charged current (CC) scattering, A31 may be determined to within about 10- 
20% of the current best fit value. In addition, by measurements in the electron appearance 
channel, MINOS may provide a possibly improved upper bound on sin^ 2^^i3, by a factor of 
1.5 to 2. In Europe, the two experiments planned in this category are the Imaging Cosmic 
And Raire Underground Signals (ICARUS) project [25], a 2.35 kT liquid Argon de- 
tector, and the Oscillation Project with Emulsion-tRacking Apparatus (OPERA) 
[26], a 1.65 kT emulsion cloud chamber which will ride the 732 Km CERN to Gran Sasso 
baseline and will be powered by the CNGS beam. The beam energy is higher, resulting in 
neutrinos which are kinematically capable of producing r leptons (< Ej, >~ 17 GeV). In 
addition, these detectors can identify muons and electrons. It is thus anticipated that in 
addition to an improved precision in A31, an improvement in the current bound on sin^ 2^13 
may be possible at the same levels as anticipated with MINOS. 

• Superbeam experiments utilise the same basic principle used in conventional beam experi- 
ments, but incorporate substantial technological improvements. This includes higher power 
beams and the idea of an "off-axis" location for the detector. One of the planned projects 
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is the 295 Km Tokai to Kamioka (T2K) project [27], with Super-Kamiokande (SK) 
as the far detector of total mass 50 kT and < >~ 0.76 GeV. Similarly, the NuMI OfF- 
Axis fe Appeeirance experiment (NOi/A) [28] is planned for location in the US, with a 
probable 810 Km baseline terminating in a 30 kT calorimeter and < >~ 2.2 GeV. Both 
experiments primarily aim at heightened sensitivity to sin^ 2^^i3 via the electron appearance 
channel. It is anticipated that the upper bound on this parameter will be improved by a 
factor of four over a five year running period. 

As discussed in [23], the combination of conventional and superbeam experiments over the 
next 10-12 years will improve the precision on A31 by an order of magnitude, while the 
improvement in sin^ 2^23 will be much more modest, i.e by a factor of about two. 

• Planned experiments in the reactor category [29] include KASKA in Japan [30], one in 
Diablo Canyon, USA [31] and another in Daya Bay, China [31] and an upgraded version 
of CHOOZ [32], called Double-CHOOZ [33] in Prance. Reactor experiments detect the 

electron anti-neutrino flux by the inverse beta-decay process, and focus on improved measure- 
ments of sin^ 26*13, using a near detector to lower systematics. A factor of six improvement 
in the present upper bound on this parameter is expected. 

To summarize, the above experiments will, over the next 10-12 years, greatly improve the 
precision on A31, effect a very modest improvement in the existing measurements for sin^2^23; 
and improve the upper bound on sin^ 26*13 by a factor of two to six, depending on the experiment. 
We note that given their insensitivity to matter effects^, they will not be able to conclusively 
determine the sign of A31 and thus will not establish whether neutrino masses follow a normal 
hierarchy or an inverted one^. This may thus leave one of the major questions of neutrino physics 
unresolved over the timescale considered here (10-12 years). Besides requiring a baseline long 
enough to allow matter oscillations to develop, the resolution of this issue, in general, requires 
a detector which can distinguish the sign of the lepton produced in a CC interaction. Over a 
longer term, progress in this direction may be possible via the proposed mega-ton water Cerenkov 
detectors [34]. Even though they do not have the capability to distinguish the charge on an event 
by event basis, once enough statistics are collected (~ 2 mega-ton year exposures), it may be 
possible to use the differences in total and differential CC cross-sections between neutrinos and 
anti-neutrinos to obtain a statistical determination of the sign of A31 [35, 36, 37]. Additionally, if 
neutrino factories are built, they will be able to resolve this question in a definitive fashion [38]. 
This paper examines the possibility of resolving this issue using atmospheric muon neutrinos over 
the same time-frame as the long baseline program described above (shorter compared to the time 
scale of neutrino factories) . 

In the first part of this paper, we study the physics related to the sensitivity of the muon 
neutrino survival probability to matter. In general, P^^^ — 1- P^e - P/irj ^-nd one normally assumes 

^All baselines for the listed experiments are around or below 800 Km. 

^Combined results from T2K [27] and the NuMI OfF-Axis [28] experiment may be able to infer the neutrino 
mass hierarchy. However, first results on this would be available about 6 years after the NOi/A far detector is 
completed, i.e., around 2017. In addition, inferring the hierarchy may be complicated by ambiguities resulting from 
uncertainties in sin^ 2^13 and ^cp- 



4 



that the dependence of P^^ on matter effects arises primarily from its P^^^e component while matter 
effects in P^^ remain small. This is because and i/t have the same coherent interactions with 
the Earth's matter. In [39] recently, it was shown that contrary to expectations, the — > Vt 
oscillation probability P^^ can also undergo significant change (for instance, a reduction as high 
as ~ 70% or an increase of ~ 15%) at very long baselines (> 6000 Km) over a broad band of 
atmospheric (GeV) neutrino energies due to matter effects. Given the fact that the atmospheric 
neutrino flux is a sharply falling function of energy in the few GeV range cx E""^-^) and 
that the production of r leptons via CC is kinematically suppressed here, a direct observation 
of this effect via appearance in an atmospheric neutrino experiment may be difficult. However, 
large matter effects in P^^ can cause correspondingly large changes in P^^, and we explore the 
consequences of this in our paper. In particular, in the next section, we systematically discuss the 
inter-relation between the three probabilities P^e, P/i/i and P^^- and study the ranges of energies 
and pathlengths where they combine in a synergistic manner to give large effects. We also examine 
the problem of parameter degeneracies in these oscillation probabilities with reference to very long 
baselines. 

The remaining part of the paper studies observational consequences of these effects for muon 
survival rates with particular emphasis on resolving the hierarchy issue in an atmospheric neutrino 
setting, using a detector capable of lepton charge discrimination. Earlier studies of this were 
undertaken in [40, 41, 42] and, more recently, in [43, 44, 45]. We also note that the muon survival 
rate is a major constituent of the signal in the existing SK [2, 3] detector, the planned mega-ton 
Water Cerenkov detectors like Underground Nucleon decay and Neutrino Observatory 
(UNO) or Hyper-Kamiokande [27, 34, 46], and several detectors considered for future long 
baseline facilities [47] , for which the discussion below may be of relevance. 



2 Discussion of the Matter Probabilities : P;;,, P;^^ and F^'^ 

Analytical expressions for oscillation probabilities for neutrino propagation in vacuum and Earth's 
matter have been extensively studied in the literature [48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 
60, 61, 62, 63, 64]. The neutrino flavour states are linear superpositions of the mass eigenstates 
with well-deflned masses : 

Wa) Wi) , (1) 

i 

where [/ is a 3 x 3 unitary matrix known as the Pontecorvo-Maki-Nakag mixing matrix. 

We use the standard parametrization of U in terms of three mixing angles and a Dirac-type phase 
(ignoring Major ana phases), viz., 

U = I -C23S12 - S23Si3Ci2e*'^CP C23C12 - S23Si3Si2e*^CP S23C13 \ , (2) 

S23S12 - C23Si3Ci2e*'^CP -S23C12 - C23Si3Si2e"^c;P C23C13 

where Cij = cos 9, ij, Sij = sin^j^ and 6cp is the CP-violating phase. 

For the purpose of our discussion in the flrst three subsections here, besides the approximation 
of constant density, we set A21 = Agoi = 0. Consequently the mixing angle 612 and the CP 
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phase 5cp drop out of the oscillation probabilities. This approximation simplifies the analytical 
expressions and facihtates the qualitative discussion of matter effects. We have checked that 
this works well (upto a few percent) at the energies and length scales relevant here. However, 
all the plots we give in this paper are obtained by numerically solving the full three flavour 
neutrino propagation equation assuming the Preliminary Reference Earth Model (PREM) [65] 
density profile for the earth. We use A31 = 0.002 eV^ and sin^ 26*23 = 1 unless otherwise mentioned. 
Further, the numerical calculations assume A21 = 8.3 x 10~^ eV^, sin^ Ou — 0.27 [17] and Sqp — 0. 
The effect of varying the CP phase over the entire range (0 to 27r) will be taken up in Subsection 
2.4, where we include analytic expressions which take into account small sub-leading effects and 
discuss the associated parameter degeneracies. 

2.1 Review of P^e in matter 

We first review i/^ Ve oscillations in matter. In vacuum, the v^. oscillation probability is 

P;, = sin^ ^23 sin^ 2^13 sin^ (I.27A31L/E) , (3) 

where A31 = mg — mf is expressed in eV^, L in Km and E in GcV. In the constant density ap- 
proximation, matter effects can be taken into account by replacing A31 and ^13 in Eq. 3 by their 
matter dependent values, i.e., 

= sin^ ^23 sin^ 29"^^ sin^ (1.27A^iL/E). (4) 
Here A^ and sin 29^^ are given by 

A^i = ^( A31 cos 2^13 - A)2 + ( A31 sin 2^13)' 

sin 2^-3 = sin 2^13^ (5) 

^31 

where, 

A = 2a/2 Gf Ue E = 0.76 x 10"^ p(gm/cc) E(GcV) 
The resonance condition"^ is A = A31 cos 26*13, which gives 

^ A3I cos 2^13 

2v^GFne ^ ^ 

Naively, one would expect P™ to be maximum at E = Ereg since sin2^^"3 = 1. But this is 
not true in general because at this energy /S^^ takes its minimum value of A31 sin 26*13 and P™ 
remains small for pathlengths of L < 1000 Km. If L is chosen suitably large so as to satisfy 
(I.27A31 sin2^i3L/E) > 7r/4, then can attain values > 0.25 for sin^ 2^23 = 1- For A31 = 0.002 
eV^ and sin^ 2^13 = 0.1, one needs L > 6000 Km to satisfy the above condition. 

"^Notc that this condition is sensitive to the sign of A31. A31 > gives rise to matter enhancement in case of 
neutrinos, while for anti-neutrinos (since A —A) one gets a suppression due to matter effects. The situation is 
reversed for A31 < 0. 
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Figure 1: pL plotted vs L. Horizontal lines correspond to [joL]™^'*'*^ for different values of ^13 calculated using Eq. 7 
for p=0. 



In particular, P™ is maximum when both 



sin 2^5"3 = 1 



and 



sin^ (1.27A^iL/E) = 1 or, (1.27A;^iL/E) = [(2p + l)7r/2] 
are satisfied. This occurs when Eres = E™eak- This gives the condition [39, 40, 



[pL] 



max 
fie 



(2p + l)7r5.18 X 10^ 
tan 2^13 



Km gm/cc. 



(7) 



Here, p takes integer values. This condition is independent of A31 but depends sensitively on 
^13. p in Eq. 7 is the average density of matter along the path of travel. For trajectories passing 
through earth's core, p strongly depends on the pathlength. In Figure 1, we plotted pL vs L 
for pathlengths varying between 700 Km to 12740 Km based on PREM profile of earth density 
distribution. In this paper, the symbol p refers to the value of earth matter density obtained from 
Figure 1, for any value of L. We identify the particular values of pL which satisfy Eq. 7 with p = 
for three different values of sin^ 2^13. These occur at L ~ 10200 Km, 7600 Km and 11200 Km for 
sin^ 2^13 = 0.1,0.2 and 0.05 respectively. This identifies the baselines at which P™ is maximized. 
Additionally, the relatively wide spacing between them demonstrates the sensitivity to ^13. At and 
around the resonant energies and these baselines (depending on the value of 6*13) P™ significantly 
impacts not only P^^, but also P^,-, as we discuss below. Note that for higher values of p, the 
baselines for maximum matter effect are greater than the Earth's diameter. Hence p = is the 
only relevant value of p in this case. 
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2.2 Matter effects in P^,^ 



In vacuum we have 



PI, = cos^^i3sin2 2^23sin2(1.27A3iL/E), 



flT 

cos^ ^13 sin^ 2^23 sin^ (I.27A31L/E) 



- COS^ ^23? 



fic 



Including the matter effects changes this to 



P;:;; = cos^ ^J^gsin^ 2^23 sin^ [1.27(A3i + A + A;^i)L/2E] 
+ sin^ ^J^sin^ 2^23 sin^ [1.27(A3i + A - A^," )L/2E] 
- cos2^23P;re (9) 



A = 



V 

e 

V 



degenerate 



(a) 



A = A 



(b) 



m, (mat) 



(mat) 
m,^(mat) 



Figure 2: The flavor composition of mass eigenstates in vacuum (A = 0) and in matter at resonance (A = Arcs)- 
Tlie vacuum parameters used are sin^ 2023 = 1-0 and sin^ 2^13 = 0.1. 



Compared to P™ , these expressions have a more complex matter dependence. This occurs due 
to the more complicated change of and in the flavour content of the matter dependent mass 
eigenstates. Labeling the vacuum mass eigenstates as z/i, i>2 and u^, in the approximation where 
A21 = 0, ui can be chosen to be almost entirely Ue and i>2 to have no Ue component. Inclusion of 
the matter term A leaves z/2 untouched but gives a non-zero matter dependent mass to ui, thereby 
breaking the degeneracy of the two mass states. As the energy increases, the Ue component of 
z/™ decreases and the i/^, Ur components increase such that at resonance energy they are 50%. 
Similarly, increasing energy increases the Ue component of (and reduces the z/^, components) 
so that at resonance it becomes 50%. Thus in the resonance region, all three matter dependent 
mass eigenstates z/™, z/™ and z/™ contain significant z/^ and z/^ components. Hence, the matter 
dependent z/^ oscillation probability will depend on all three matter modified mass-squared 
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differences. In Figure 2 we show the flavour composition of the three mass states in vacuum and 
in matter at resonance. 

We are interested in finding ranges of energy and pathlengths for which there are large matter 
effects in P^,-, i.e., for which AP^,- = P™^ — P^^ is large. To this end, it is useful to examine the 
set of probability plots given in Figure 3. The plots display the variation of P™^ with neutrino 
energy E for baselines of 8000 Km, 9700 Km and 10500 Km respectively, with each of the terms 
in Eq. 9 plotted separately. P™^(1), P™^(2) and P™t-(3) in the figure refer to the first, second and 
third term in Eq. 9 respectively, with the third term plotted incorporating its negative sign. 

Additionally, the plots display the full vacuum probability, P^^. The approximate average 
density at these baselines is p ~ 4.5 gm/cc, hence one gets E^cs — 5 GeV from Eq. 6. The vacuum 
peaks, of course, shift with baseline and we see that a significantly broad peak gradually positions 
itself above the resonant energy at the baselines shown in the figure. We use this fact in (i) below. 

We show that appreciable changes in P™^ occur for two different sets of conditions, leading in 
one case to a sharp decrease from a vacuum maximum and in another to a smaller but extended 
increase over a broad range of energies. Both are discussed below in turn. 

(i) Large decrease in P™^ in the resonance region : 

As is evident in Figure 3, at energies appreciably below resonance, term 1 in Eq. 9 (i.e. the 
one with cos^ 9"^^) is very nearly equal to P^^. This is because ^^"g = ^^13, A << A31, A!^ ~ A31. 
Term 2, the sin^ term, is nearly zero. Similarly, term 3 is also very small. As we increase 
the energy and approach resonance, cos^ 9^^ begins to decrease sharply, deviating from its 
vacuum values, while sin^ 9^^ increases rapidly. However, if resonance is in the vicinity of 
a vacuum peak, which is the case at and around the baselines shown in Figure 3, then the 
decrease in the cos^ ^^5.3 term has a much stronger impact on P™^ than the increase in the 
sin^ ^^"3 term, since the latter starts out at zero while the former is initially close to its peak 
value (~ 1). As a result, P™^ falls sharply. This fall is enhanced by the third term in Eq. 9, 
which is essentially 0.5 x P™ (which is large due to resonance), leading to a large overall 
drop in P™^ from its vacuum value. Note that the requirement that wc be at a vacuum peak 
to begin with forces AP^,- to be large and negative , with the contributions from the first 
and the third term reinforcing each other. 

The criterion for a maximal matter effect, Eres — Epeak) leads to the following condition: 

[pL]'i^^ ~ (2p + 1) TT 5.18 X 10=^ (cos 2^13) Km gm/cc. (10) 

Unlike Eq. 7, which has a tan 2^13 in its denominator, Eq. 10 has a much weaker dependence 
on 9i3. 

Prom Eq. 8 and Eq. 9, we write 

pm pv 

sin^ 2^23 [ - cos^ ^13 sin2(1.27A3iL/E) 
cos' ^5^3 sin2[1.27(A3i + A + A^i)L/2E] 
sin' ^5^3 sin2[1.27(A3i + A - A^JL/2E] 
sin' 9^s cos' ^5^3 sin'(1.27A^iL/E) ] (11) 



AP 



+ 
+ 
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Figure 3: The variation of P^r in matter (P™^) ^-nd in vacuum (P^^) with neutrino energy E for three different 
basehnes of 8000 Km, 9700 Km and 10500 Km respectively. Here we have used an approximate average density 
~ 4.5 gm/cc. Each of the terms in Eq. 9 are also shown in the three plots: P™^(1) is term 1, ¥^^{2) is term 2 and 
P™^(3) is term 3 respectively. 

Incorporating the Eres — Ep^^^^ condition (Eq. 10) leads to 



cos 



sin 2^13 (2p + l) 



TT 



(12) 



We note that, in general, AP^^ (at Eres — ^^eak ) will larger for higher values of both p 
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Figure 4: P^r and P^^ plotted as a function of neutrino energy, E (in GeV) in presence of matter and in vacuum 
for both signs of A31 for two different baseline lengths, (a) for L = 9700 Km and (b) for L = 7000 Km. These 
plots use A31 = 0.002 eV^ and sin^ 26*13 = 0.1. 



and 613. For p = 1 and sin^ 2613 = 0.1, Eres — E^eak occurs at ~ 9700 Km (from Eq. 10) and 
AP^^ ~ -0.7 (from Eq. 12). For p = 0, Eq. 10 gives L^^ ~ 4400 Km for sin^ 2^13 = 0.1. 
However, AP^.^ is roughly one-tenth of the p = 1 case. In general, for a given baseline, the 
choice of an optimal p is also dictated by the constraint that the vacuum peak near resonance 
have a breadth which makes the effect observationally viable. Note that for sin^ 29i3 = 0.05 
and 0.2, Eq. 10 gives the distances of maximum matter effect as ~ 9900 and 9300 Km for 
p = 1. Because of the weaker dependence on 613 here compared to P™ (Eq. 7), the distances 
for various values of 6*13 are bunched together in the vicinity of 9500 Km. 

In Figure 4(a), we show all three matter and vacuum probabilities for 9700 Km. In these 
plots A31 is taken as 0.002 eV^ which gives Eres — Ep^^^j^ at 5 GeV. The middle panel of 
Figure 4(a) shows that near this energy P™^ (~ 0.33) is appreciably lower compared to P^^ 
(~ 1). Thus the drop due to matter effect is 0.67, which agrees well with that obtained in 
the paragraph above using the approximate expression Eq. 12. 
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Figure 5: The plot shows sensitivity to 6*13 in case of P^r for a baseline length of 9700 Km. The value of A31 is 
taken to be 0.002 eV^ 

In Figure 5 we show the 613 sensitivity of P™^ at 9700 Km^. In particular, at Eres — ^peak 
the strong dependence on ^13 is also governed by Eq. 12 above. Thus we have significant 
change in P™^ due to matter effects, even for small values of ^13. This is illustrated in 
Figure 5, where we see that, even for as small a value of sin^ 2^13 = 0.03, matter effects at 
maximal resonance cause a change in P™^ of about 30%. This amplification of matter effects 
at resonance occurs for similar pathlengths for all allowed values of 613, as given in (Eq. 10). 
This is to be contrasted with the case of — > Uf. oscillations, where the observation of 
resonance amplification is possible only for 6is close to the present upper bound. Actual 
observation of these maximal resonance matter effects in the i/^ — > Ur oscillations will be 
very difficult, because of the smallness of cross-sections and the difficulty in reconstructing 
the T. With a very high intensity source, such as a neutrino factory and a specialized r 
detector, it may be possible. A study of this will be reported elsewhere [74]. 

(ii) Increase in P™^ away from resonance : 

We now discuss the second scenario for which P™^ can differ appreciably from P^^. This 
happens away from resonance, and is evident in Figure 4(a) (central panel) in the energy 
range 7.5 GeV - 15 GeV. This effect is an enhancement rather than a drop, i.e., AP^,- is 
now positive, unlike the previous case. 

AP^e is small in most of the latter part of the energy region under consideration and does not 
contribute in an important way overall (Figure 4(a) (top panel)). The dominant contribution 
to this enhancement arises from the sin^ term in P™^ (Eq. 9) which is large for E >> Ej-es- 

^P'^T develops good sensitivity to 6*13 only at around L = 9700 Km. As shown in the middle panel of 4(b), even 
at L— 7000 Km, its sensitivity to matter effects is relatively poor [59]. 
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Since (A31 + A — A^) ~ 2 A31 for these energies, we obtain a enhancement (~ 15%) which 
follows the vacuum curve. The difference between the two curves, vacuum and matter, largely 
reflects the difference between the cos^ 613 multiphcative term in the vacuum expression Eq. 8 
and the sin^ 6^ multiplicative term in Eq. 9. While this effect is smaller compared to the 
effect discussed in (i) above, it occurs over a broad energy band and can manifest itself in 
energy integrated event rates. 

2.3 Matter effects in P^^ 

In vacuum P^^ is given by 



pv _ 1 pv pv 

= 1 - cos^ sin^ 2^23 sin^ (I.27A31L/E) 

- sin^^23sin2 2^i3sin2(1.27A3iL/E) (13) 
Including the matter effects changes this to 

pm _ 1 _ pm _ pm 

= 1 - cos^ e^^sin^ 2^23 sin^ [1.27(A3i + A + A;^i)L/2E] 

- sin^ ^5^sin^ 2^23 sin^ [1.27(A3i + A - A^i)L/2E] 

- sin^^23sin^2^r3sin^(l-27A;jlL/E) (14) 

The deviation of PJ^ from P^^ clearly results from the combined effects in P™^ and P™ . In 
order to quantify the extent of deviation, we define, 

AP^^ = -AP^,-AP^e (15) 

Below wc illustrate the various conditions which can give rise to a significant change in P^^ 
due to matter effects arising in both P^,- and P^e '■ 

(a) Lcirge, negative AP^,- and positive AP^e 

Large and negative AP^^ corresponds to the case (i) discussed in Subsection 2.2. In this 
case, AP^e is positive, so the signs of the two changes are opposite and hence the two terms 
(Eq. 15) do not contribute in consonance. However, the resulting increase in P^^ is still 
significant (~ 20%), given the magnitude of the change (~ 70%) in P^,-- This is visible in 
the bottom panel of Figure 4(a), in the energy range 4-6 GeV. 



(b) Positive AP^^ and small AP^^c 

This case corresponds to a significant drop in P™^, which is seen in Figure 4(a) (bottom 
panel) in the energy range ~ 7 — 15 GcV. The enhancement in P™^ (corresponding to the 
case (ii) discussed in Subsection 2.2) is reflected in the decrease in PJ^'^ compared to its 
vacuum value. 
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Figure 6: Figure (a) and (b) depict P^^ in matter plotted against the neutrino energy, E(in GeV) for different 
values of 6*13 and two baseline lengths, viz, 7000 Km and 9700 Km, respectively. 



(c) Small AP^^ and large, positive AP^e 

This situation occurs when a minimum in the vacuum value of P^,- resides in the proximity 
of a resonance. The condition for a minimum in P^,- is I.27A31L/E = pvr. In this region, the 
rapidly changing sin^ 9^ and cos^ 9f!^ increase P^t from its vacuum value of to about 0.1. 
At the same time, P^e undergoes an increase of 0.3, thus leading to a net change of P^^ = 0.4 
(the three panels of Figure 4(b)). Note that the above condition corresponds to a vacuum 
peak of P^^ and the 40% drop of its value makes this energy range (4-10 GeV) suitable for 
searching for matter effects. Substituting E as Ercs gives the distance for maximum matter 
effect in P^u as 



[pL];rr - P ^ ^10^ (cos 2^13) Km gm/cc. (16) 

For p=l, this turns out to be ~ 7000 Km^. This effect [59] is shown in the bottom panel of 
Figure 4(b). The large (40% at its peak) drop in P™^ seen in this figure derives its strength 
mainly from the resonant enhancement in P™ . 

Defining AP^^ = P^^ — PJ^^ and incorporating the condition that Ej-es — E^eak of P^t^ (Eq. 16) 
leads to 

- 0.25 sin^ [sin 2^i3P7r]. (17) 

The ^13 sensitivity of P™^ for L = 7000 Km and for L = 9700 Km is shown in Figure 6. In 
Figure 6(a), we see that for L = 7000 Km, the maximum 613 sensitivity is in the energy range 

^Note that this length is close to the "magic baseline" [68, 73]. 



AP ~ 



sm 



sin 2^i3p 
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4-10 GeV. Our ability to observe this drop in P 
observability may be stated as 



depends on the statistics. The condition for 3ct 



N^(^13 = O)-N^(013)>3VN^(013). 



Figure 7 depicts the sensitivity of P™^ to 6^23 for the same distances, showing a similar inverse 
relation between the survival probability and the value of ^23- Note that, for these baselines, P^^ 
changes by as much as 20%, for the currently allowed range of ^23- A criterion similar to Eq. 18 
can be applied to these sensitivities in situations where their observability is feasible. This is likely 
to happen in long baseline experiments with high luminosity sources. 

The width of the effects discussed above for P^^ is significant, ranging from 4 — 6 GeV in case 
(a), 7—15 GeV in case (b) (Figure 4(a)) and 4 — 10 GeV in case (c) (Figure 4(b)). We have 
checked that they persist over a range of baselines (6000 - 10500 Km), making them observationally 
feasible, as is demonstrated later. 

In general the resonance has a width, and this fact affects observability. Below we give an 
expression for AP^^ which incorporates the resonance width and thus quantifies the deviation 
between Eres and ^p^ak which can exist while still keeping the effect observable. To include the 
width of the resonance, we write A = A31 (cos 2^13 + qsin2^i3), where q varies from -1 to 1. With 
this parametrization. 



AP 



MM 



— Q+ sin^ 
Q_ sin^ 



A31L / . 
1.27^^ I 1 - sm2^i3 



1.27^— I 1 + sm 26'i3 ^ 
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- l/[4(q2 + l)]sin2 
+ sin2(1.27A3iL/E) 



A31L 



1.27^ ( sin 2^^i3Vq' + l 



(19) 



Here, Q± = (Vq^ + 1 ± q)/ (2-\/q^ + 1). In obtaining Eq. 19 we have approximated cos 26'i3, cos^ 6'i3 ~ 1. 

2.4 Degeneracies in the determination of oscillation parameters at 
very long baselines 

In our analytic discussion of matter effects on oscillation probabilities, we have used approximate 
expressions with the solar mass squared difference A21 set to zero. While this is adequate for a 
broad understanding of matter effects, it is interesting to look at a more accurate picture with 
A21 7^ and analyze the issue of parameter degeneracies associated with the inclusion of sub- 
leading effects. In this situation, the determination of neutrino mass and mixing parameters is 
complicated by the presence of degeneracies in the oscillation probabilities, which are inherent in 
a three generation analysis due to the presence of the non-zero CP phase 6cp- The degeneracies, 
extensively studied in the literature for baselines less than 3000 Km, are the {6cp, On) ambiguity, 
the sign(A3i) or mass hierarchy degeneracy and the (^23; 7r/2 — ^^23) or atmospheric angle 
degeneracy, combining to give an overall eight-fold degeneracy [67, 68, 69, 70]. It is relevant to 
discuss briefly the effect of parameter degeneracies in the context of very long baselines, as are 
considered in this paper. 
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Figure 8: (a) CP trajectory orbits in ( P^e, P 
(I.27A31L/E = 3tt/2). Orbits with different values of ^13 do not overlap, showing that the ((5cp,6'i3) degeneracy 
is resolved for these values of L and E. (b) CP trajectory plots in (P^e: Pfj.e) biprobabihty space for two different 
values of ^13 and both signs of A31. Orbits with positive and negative A31 do not coincide, showing that the 
sign(A3i) degeneracy is resolved for this L and E till ^13 as low as sin^ 2^13 — 0.001. 
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The analytic treatment of degeneracies is based on approximate expressions for the oscillation 
probabilities in matter of constant density. These are computed by series expansions in the small 
parameters - the solar mass squared difference A21 and the mixing angle 613. CP trajectory orbits 
in biprobability space are widely used for depicting degeneracies, and are conventionally plotted 
using the analytic expressions. For the purpose of the biprobability calculations here, analytic 
expressions become progressively inadequate beyond 4000 Km, or more precisely, for values of L 
and E such that L/E > 10^ Km/GeV. Additionally, the small ^13 expansion also fails for relatively 
large values of ^13 (close to the present CHOOZ bound). In view of this, we have plotted the CP 
trajectories for a sample long baseline using the full numerical solution of the evolution equation 
with earth matter effects. However, the analytic expressions remain useful for a qualitative un- 
derstanding of the features and interdependence of parameter degeneracies, and we use them for 
this purpose whenever necessary. 
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Figure 9: Same as Figure 8(a) and 8(b) but in (P^^, P^^) biprobability space. In Figure (b) the sign(A 
degeneracy is present for sin^ 29i3 — 0.01. 
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The approximate analytic expressions for oscillation probability between two flavors a, j3 are 
of the form [61, 69] 



a/3 



Xq,/} cos 5cp + sin 5cp + Z 



a/3! 



(20) 



where Xq,^, Yap and Zap are functions of the neutrino mass squared differences and mixing param- 
eters but independent of the CP phase ^cp- Note that Zq^ contains the dominant contribution to 
the probabilities given earlier (Eq. 3, Eq. 4, Eq. 8, Eq. 9, Eq. 13, Eq. 14). It can be shown that 
the generic form of a CP trajectory in biprobability space (the orbit traced in (P^^, ^ap) space as 
5cp varies from to 27r) is an ellipse [69], which collapses to a line under certain conditions. 
Here P^p is the CP conjugated (anti- neutrino) probability. Specific expressions for P™ , P™^ and 
P™^ may be found in [64]. The small A21, small 6'i3 series expansions (to second order in these 
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parameters) are compact and hence easier to analyze for degeneracies. Below we list the expressions 
for the coefficients Xq,^ and Yap in this approximation for P™ and P™^ : 



X„e = 2a sin 613 sin 26*12 sin 26*23 cos A 



Y^e = — 2q; sin 6*13 sin 2^12 sin 26*23 sin A 



sinAA sin(A - 1)A 
T (A - 1) 
sinAA sin(A - 1)A 



A 



(A-1) 



X 



A-1 



2a sin 6*13 sin 26*1 5 sin 26*23 cos 26*23 sin A 



'12 



23 ' 



723 ■ 



. -f . . sin A A - . ^ , 
X [A sm A J — cos(A - 1) A] 



Y^^ = 2a sin ^13 sin 2^12 sin 2^23 sin A 



A 

sinAA sin(A - 1)A 
~k (A-1) 



(21) 



where A = I.27A31L/E, A = A/IA31I, a = IA21I/IA31I. The above expressions are for the normal 
mass hierarchy. For the inverted hierarchy, the transformations A —A, a —a and A — > —A 
are required. The coefficients for P™^ are given by X^^ = — X^e — X^^, and similarly for Y^^. 
Note that Y^^ = and hence the survival probability is independent of the CP-odd term. Next 
we discuss in turn each of the possible degeneracies. 
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Figure 10: Same as Figure 8(a) and 8(b) in (P^r, P/jt) biprobability space. 



The (^cP; ^13) degeneracy arises when different pairs of values of the parameters 6cp and ^13 
give the same neutrino and anti-neutrino oscillation probabilities, assuming other parameters 
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to be known and fixed. This may be expressed as 

Pa/3('^CP, ^'13) = Pa/3 ((^cp, ^13) (22) 

This ambiguity manifests itself in the intersection of CP orbits with different values of 61^. 
The intersection points indicate the different values of 6cp and 613 for which the above Eq. 22 
are satisfied. For values of L and E such that A = n7i/2, Eq. 21 predict that either sin^cp 
or cos^cp drops out of the expression for P™, reducing the CP trajectory to a straight 
line. Further, it can be shown that the ((5cp,^i3) ambiguity reduces in this case to a simple 
(5cp,7r — 6cp) or ((5cp,27r — 6cp) ambiguity which does not mix different values of 6*13 [68]. 
This fact holds true at long baselines also, as is depicted in Figure 8(a) for L = 9700 Km, 
E = 5.23 GeV (giving A = 37r/2). Here the CP orbits appear as widely separated narrow 
ellipses, showing that the {6cp, 6*13) ambiguity is effectively resolved''. CP orbit diagrams for 
P™^ and P™^ are given in Figure 9(a) and 10(a), illustrating a similar lifting of the degener- 
acy involving 613 in both cases. The approximate expressions (Eq. 21) show that the muon 
neutrino survival probability is a function only of the CP-even term cos^cp, while P™^ only 
depends on sin^cp when 623 is maximal, so their CP orbits resemble straight lines instead 
of ellipses. 




(a) (b) 

Figure 11: (a) CP trajectory plots in (P^e, P/ie) biprobability space for different values of 613 and complementary 
values of 623. (b) Same as Figure 11(a) but in (P^^;^, P^/^) biprobability space. 



''Here we have given a theoretical discussion of degeneracies in probabilities, and therefore not taken into account 
possible error bars in the experimental results for event rates, which, if large enough, may cause points on different 
9i3 orbits to overlap. 
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The mass hierarchy degeneracy occurs due to identical solutions for P and P for different 
pairs of 5cp and ^13 with opposite signs of A31 (again fixing other parameters): 

P„^(A3l > 0,5cP,^13) = Pa;3(A31 < 0, (5^p, ^^3) 

PapiAu > 0, 6cP, ^13) = Pa;3(A31 < 0, 5^p, ^^3) (23) 

Note that a combined effect of the (5cp,6'i3) ambiguity and the sign(A3i) degeneracy gives 
rise to a four- fold degeneracy. From the expression for P™ , it has been determined [68] that 
the condition for this ambiguity to be resolved is 

. 2a tan 6^23 sin 26112 sin AA 
sm2t/i3 > 24 

* |- sin(l-A)A sin(l+A)A i ^ ' 

This places constraints on ^13 as well as the baseline, which must be large enough for the 
denominator to be large (thus weakening the constraint on 613). Figure 8(b) gives the orbit 
diagram for the energy and baseline earlier discussed. It is seen that the orbit ellipses for 
positive and negative A31 are well-separated for sin^ 26'i3 = 0.001. Thus the degeneracy is 
lifted even for very small values of ^13. From Figure 10(b), it is clear that P™^ is also free 
from this ambiguity for large baselines. However, P™^ shows a mass hierarchy degeneracy 
for sin^26'i3 = 0.01, as seen in Figure 9(b). 

Currently 6^23 is determined from P^^ in atmospheric (SK) and accelerator (K2K) experi- 
ments. Since this is a function of sin 26*23, these measurements cannot differentiate ^23 from 
7r/2 — 6^23- If the probability is a function of sin^ ^23 or cos^ 6*23 (e.g. the leading contributions 
in P^c and Per respectively), then for a fixed value of 6*13, Po/3(6'23, 6*13) 7^ Po/3(7r/2 — ^23, 6*13). 
However, different values of 613 and the CP phase 6cp can make them equal^, leading to an 



^This degeneracy may be present even if Sqp ~ 0. 
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ambiguity in the determination of the latter two parameters known as the {92^.,'k/2 — 623) 
degeneracy: 

Pap{023, 5cP, ^^13) = P„5(7r/2 - 92S, 5^p, ^^'13) 

Pap{e23, 5cp, ^13) = Pa/3(7r/2 - ^23, 5^P, ^ia) (25) 

The presence of this ambiguity along with the two previously discussed degeneracies will 
give rise to an eight-fold degeneracy [68] in such probabilities. However, if a probability is 
a function of sin^ 26123 (e.g. P^r), then for a fixed 6*13, Pa/3 (6*23, ^13) = Pa/3 (vi"/2 - 6123, 6*13 ) 
to start with. Therefore, different values of ^13 and 6cp give rise to different values of the 
probability, and the (6*23, 7i/2 — 623) degeneracy will not lead to any further ambiguity in the 
determination of ^13. So the total degeneracy in this case is four-fold^. 

The CP trajectories for P™ , PJ^^ and P™^ with complementary values of ^23 and our sample 
L and E are given in Figure 11(a) and (b) and Figure 12. From Figure 11(a) and (b), we see 
that the different ^23 orbits do not intersect, i.e. this degeneracy is resolved in P™ and P™^ 
for a long baseline like the one discussed, even though in principle it could exist for these 
probabilities as they are functions of sin^ 6*23. However, in the case of P°!^, as discussed above, 
this degeneracy is absent since its approximate matter expression contains only sin^ 2^23 in 
its dominant terms. This feature is evident in Figure 12. We see that all points on the orbits 
with complementary ^23 and equal ^13 coincide, so no two independent values of 5cp can 
exist which give the same probability. 

• It can also be observed from the analytic expressions for X^^ and Y^p (Eq. 21) that the magic 
baseline scenario [68, 73] discussed for P™ is equally valid for P™^ and P™^ when ^23 = 7r/4. 
This method of degeneracy resolution requires the baseline to satisfy AA = rni, in which 
case all coefficients of 5cp vanish. The energy and parameter-independent condition on L 
becomes pL — 32532 Km gm/cc. From the pL vs L curve (Figure 1), it is seen that this 
corresponds to L ~ 7600 Km. An identical analysis applies to P™^ and P™^ with maximal 
^23, with only a small correction if ^23 is non-maximal. 

In conclusion, we find that a study of probabilities at a baseline and energy like the one dis- 
cussed demonstrates, in general, a breaking of most of the parameter degeneracies which can 
confuse the determination of ^13 from oscillation measurements. For P^e, this is true for all three 
classes of degeneracies mentioned above. In the case of the muon survival probability. Figure 9(b) 
shows that sign(A3i) may still be undetermined for specific values of ^13 and the CP phase 5cpi 
for which the same value of {P nn^P are obtained with opposite signs of A31. For example, for 
sin^2^i3 = 0.01, we have checked that 5cy> — 0.3057r and (5cp = 0.3377r give the same point in 
(P^^,P^^) space for positive and negative A31 respectively for L = 9700 Km and E = 5.23 GeV. In 
P^,^. all degeneracies in\'ohiiig a measurement of ^13 and 5cp are lifted for the energy and baseline 

^Note that probabilities that are functions of sin^ 2023 : though only 4- fold degenerate with respect to ^13 and 
Sq-p, cannot ascertain whether 623 < 7r/4 or 623 > 7r/4. This information can be obtained from a comparison 
of probabilities which are functions of sin^ 623 and cos^ 623 (for e.g., the golden and silver channels discussed in 
[71, 72]). 



21 



discussed. Also, we note that the maximum total degeneracy of P^^ is four-fold even in principle, 
since its analytic expression in matter (to second order in the expansion parameters) is a function 
of sin 2^23- 



3 Determining the Mass Hierarchy via Atmospheric in 
a charge discriminating Detector 

In the discussion in Sections 2.1-2.3, we have shown that large matter effects in neutrino oscillations 
are not necessarily confined to —>■ or i/^ —>■ fr conversions, but can be searched for in —>■ 
oscillation and — > z/^ survival probabilities. We have examined their origin by studying the inter- 
relations of all the three matter probabilities, P™ , P™^ and P™^, and identified baseline and energy 
ranges where they act coherently to give observationally large effects. The effects discussed are 
strongly sensitive to the sign of A31, as is apparent from the plots shown in Figure 4. It is useful to 
recall Eq. 15 and emphasize that contrary to what one generally assumes, AP^^ is not necessarily 
negative, i.e. P™^ is not always less than or equal to P^^. This would be true if large matter 
effects resided only in P™ , since AP^c is positive over the relevant range of energies and baselines. 
However, as shown earlier, matter effects can not only lead to a substantial enhancement of P™ , 
but can also cause an appreciable drop or rise in P™^, i.e. AP^^ can be significant. Depending 
on the extent of this, P™^ may he larger or smaller than P^^. In this section, we focus on the 
detection of matter effects in muon and anti-muon survival rates in atmospheric neutrinos. 

Atmospheric neutrinos, while not a controlled source in the sense of the beam experiments 
described above, offer a compensating advantage: a very broad L/E band extending about 5 
orders of magnitude (1 to 10^ Km/GeV, with neutrino energies, Ej, ~ 1 GeV and above, L from a 
few Km to about 12500 Km). The longer baseline lengths allow matter effects to develop, and offer 
possibilities for determination of the mass hierarchy using a detector capable of identifying muon 
charge. Other physics possibilities with a charge discriminating detector using either atmospheric 
neutrinos or a neutrino beam as source have been discussed in [38, 43, 75, 76, 77]. 

Prior to giving the results of our calculations, we briefly describe some of the inputs we have 
used. The prototype for our calculations is a 100 kT Iron detector, with detection and charge 
discrimination capability for muons provided by a magnetic field of about 1.2 Tesla. Sensitive 
elements arc assumed to be Glass Spark Resistive Plate Chambers (RPC). We have assumed a 
(modest) 50% efficiency of the detector for muon detection. This incorporates the kinematic cuts 
as well as the detection efficiency. The L/E resolution will depend on event kinematics and on 
muon track detection capabilities^" . In the calculations presented here, we have used the Bartol[78] 
atmospheric flux and set a muon detection threshold of 2 GeV. These specifications have been 
culled from the MONOLITH [79] and INO [80] proposals and hence are realistic to the best of our 
knowledge. The main systematic errors in this experiment are those that arise in the determination 

^'^For our study, we do not use any resolution function. For the most part in what follows, we draw our conclusions 
based on event rates summed over fairly large ranges in L and/or E. 



22 



F (C-,(A/] =^ 

1— 1 VJC V 1 — 7^ 




3-5 


5-7 


7-10 


9-10 


L (Km) JL 


N t N 

■'■''mat) ■'■'vac 


N t N 

'-'mat) ■'■'vac 


N t N 

■'■'mat) ■'■'vac 


N t N 

■'■'mat) ■'■'vac 


N ^ N 

^'mat) ^'vac 


2000 - 4000 


99, 102 


37, 41 


11, 12 


23, 23 


170, 178 


4000 - 6000 


55, 59 


92, 93 


17, 19 


3, 3 


167, 174 


6000 - 8000 


71, 71 


47, 48 


45, 45 


10, 12 


173, 176 


8000 - 9700 


49, 49 


47, 48 


27, 25 


22, 23 


145, 145 


9700 - 10500 


25, 26 


20, 21 


6, 5 


12, 13 


63. 65 


10500 - 12500 


60, 58 


54, 55 


10, 11 


27, 26 


151, 150 


2000 - 12500 


359, 365 


297, 306 


116, 117 


97, 100 


869, 888 



Table 2: Number of /U+ events in matter and in vacuum in restricted bins of E and L for A31 = 0.002 eV^ and 
sin^ 26113 = 0.1, sin^ 26I23 = 1-0. 

of the energy and the direction (and hence pathlength) of the neutrino in the initial state. As is 

true in all neutrino experiments using extra terrestrial sources, the statistical errors are expected 
to dominate the systematic errors. So, in this discussion, we have not taken systematic errors into 
account. 

In order to demonstrate the important qualitative features of our results, we have selectively 
provided both tables as well as plots of event rates versus L and L/E for positive A31. Atmospheric 
neutrinos offer the advantage of being able to appropriately select ranges in L and E from a large 
spectrum of L and E values. 

An approximate feel for where significant matter effects are likely to show up in muon survival 
rates can be obtained using our earlier discussion in Sections 2.1-2.3, especially that which leads to 
Eq. 6, Eq. 7, Eq. 10 and Eq. 16. In particular, the resonance condition (Eq. 6) gives the following 
constraint on the product of density and energy 

pEres = 1.315 X 10^ A31 COS 2^13 GeV gm/cc (26) 

From Eq. 26, using the value A31 = 0.002 eV^ and approximating to the case of a constant average 
density p for any given baseline, one obtains the expression^^ 

pEres ^ 25 GeV gm/cc. (27) 

In Figure 13, we plot the average density, p as a function of the pathlength, L for the earth. A 
rough measure of Ercs for any given baseline may thus be obtained using this plot and Eq. 27. This 

^^Note that the resonance condition is not very sensitive to the value of ^13, since the dependence is through 
a cosine. However, it does depend on what value of A31 we use. For example, for A31 = 0.002 eV^, pE^es = 
26.3 X cos 26*13, which gives pE^s = 24.96 for sin^ 26*13 = 0.1 and pErcs = 26.18 for sin^ 26*13 = 0.01. However, 
for A31 = 0.0015 eV^, pE^s = 19.7 x cos 26*13, giving pE^s = 18.7 for sin^ 26*13 = 0.1 and pE^s = 19.6 for 
sin^ 26*13 = 0.01. 
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I— 1 VJC V 1 — 7^ 

L fKm) JL 




3-5 

^"mat; ^"vac 


5-7 

N , N 

^"mat; ^"vac 


7-10 


9-10 

^"mat) ^"vac 


2000 - 4000 


229, 233 


99, 93 


31, 28 


55, 53 


414, 407 


4000 - 6000 


134, 139 


191, 215 


42, 45 


10, 8 


377, 407 


6000 - 8000 


168, 169 


115, 113 


81, 109 


21, 30 


385, 421 


8000 - 9700 


120, 118 


129, 115 


59, 63 


43, 59 


351, 356 


9700 - 10500 


63, 62 


39, 51 


24, 13 


29, 32 


155, 158 


10500 - 12500 


136, 138 


118, 138 


29, 28 


70, 69 


353, 373 


2000 - 12500 


850, 859 


691, 725 


266, 287 


228, 251 


2035, 2122 



Table 3: Number of fi events in matter and in vacuum in restricted bins of E and L for A31 — 0.002 eV^ and 
sin^ 26*13 = 0.1, sin^ 2^23 = 1.0 (see text for details). 

is also useful in understanding the broad features of the tables representing our actual calculations, 
and in selecting baseline and energy ranges for closer scrutiny, even though the constant density 
approximation fails once baselines are very long. 




356 2000 4000 6000 8000 10000 12000 



L (Km) 

Figure 13: The average density p for each baseline L, obtained from the PREM density profile, plotted vs L. 

We now proceed with the discussion of our event rate calculations which, as stated earlier, 
have been performed by solving the full three flavour neutrino propagation equation using PREM 
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[65] density profile of the earth. We have scanned the energy range 2-10 GeV^^ and the L range 
of 2000-12500 Km for evidence of matter effects. We have assumed A21 = 8.3 x 10"^ eV^, 
sin^ 612 = 0.27[17], 6cp = and an exposure of 1000 kT-yr for the tables and all plots. Note that 
for baselines > 10500 Km, passage is through the core of the earth [81]. 

In Table 2, we show the fi~^ event rates in matter and in vacuum for various energy and baseline 
bins. As expected, matter effects are negligible for anti-muons if A31 is positive. 

L = 6000 to 9700 Km, E = 5 to 1 GeV 



=1 




5 - 



I . \ > 1 . \ . 1 

6000 7000 8000 9000 

L(Km) 

Figure 14: Atmospheric muon and anti-muon events plotted against L for energy range of 5 GeV to 10 GeV and 
baselines between 6000 Km to 9700 Km. The numbers in the legend correspond to the integrated number of muon 
and anti-muon events for the restricted range of L and E in matter and in vacuum for a given value of sin^ 2^13. 

Table 3 shows the atmospheric event rates for the same bin choices, but for muons. We note 
that while there are visible differences between matter and vacuum rates, they are not always 
significantly large. A careful selection of energy ranges and baselines is necessary to extract a 
statistically significant signal. Our plots below refiect these selections. In particular, in Table 3, a 
~ 4(7 signal for matter effects can be extracted from the energy bins 5-10 GeV and the L bins 6000- 
9700 Km (shown in bold text in the table), as discussed below. Consolidating the event rates over 
large bins in energy and baseline will lead to a dilution, or possibly a wash-out of the signal, as is 
evident from this table. For example, the right-most vertical column shows the energy integrated 
events for the range 2-10 GeV and the last row lists the events integrated over the 2000-12500 Km 
baseline range. In both cases, one notes a dilution of the signal for matter effects. We also note 
that for neutrinos passing through the core [81], the difference between the matter and vacuum 
rates is not appreciable except in the energy range 3-5 GeV for these values of parameters'^ (Table 

^^Higher values of energies are also possible, but the falling flux factor considerably diminishes the event rates. 
Therefore we take E upto 10 GeV only. 

^^Varying sin 623 and keeping sin 2^13 fixed does not change the muon event rates significantly. For instance, 
for sin^ ^?23 = 0.6 and sin^ 2^13 — 0.1, we get 116 events in matter as compared to 139 events in vacuum in the 
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3, sixth row). These effects may be looked for in neutrino factory experiments. 

L = 6000 to 9700 Km, E = 5 to 1 GeV 
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Figure 15: The total event rate for muons and anti-muons in matter and in vacuum plotted against Logio(L/E) 
for the restricted choice of L and E range (see Figure 14) . The numbers in the legend correspond to the integrated 
number of muon and anti-muon events for the restricted range of L and E in matter and in vacuum for a given 
value of sin^ 26i3 . 

In Figure 14 we show event rates for muons and anti-muons versus L for the basehne and energy 
bins mentioned above (L = 6000-9700 Km, E = 5-10 GeV). We have assumed A31 = 0.002 eV^ 
and sin^ 2^13 = 0.1 for this plot. The effect is large for this choice of energies and baselines because 
it is a combination of the two effects visible in the bottom panels of Figure 4. The fall in P™^ at 
9700 Km between 6-15 GeV (which persists over a range of baselines) obtains a large contribution 
from P™^ while the effect shown at 7000 Km arises primarily due to P™ . Both work to lower the 
muon survival rate below its vacuum value. In contrast to an expected vacuum oscillation rate of 
261 events, one expects 204 events in matter if the sign of A31 is positive. 

That we have indeed chosen the bins for which matter effects and sensitivity to the mass 
hierarchy are maximum, can be demonstrated by a computation of the sensitivity with which the 
expectation for matter effects with Normal Hierarchy (NH) differs from that for Inverted Hierarchy 
(IH) or for vacuum oscillations. To find the sensitivity, we define 

Nnh - N, 



I^^NH other I /'oo^ 

C"NH-othcr — 7^= ' 1^°/ 

V J-^NH 

where Nother = Nm when computing the sensitivity to mass hierarchy (Tnh-ih and Nother = Nvac for 
the sensitivity to matter effects aNH-vac- We split the 24 bins in Table 3 into two sets, with fixed 

energy bin E = 3 - 5 GeV and baseline range L — 10500 - 12500 Km. However, as expected, decreased values of 6*13 
do diminish the statistical significance of the signal. For sin^ 623 = 0.4 and sin^ 2^13 = 0.05, we get 132 events in 
matter while in vacuum we get 140 events in the energy bin E = 3 - 5 GeV and baseline range L = 10500 - 12500 
Km. 
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values of parameters A31 = 0.002 eV^, sin^ 26*23 = 1-0, sin^ 26*23 = 0.1. The 4 highlighted bins form 
one set and the other 20 bins form the second. The sensitivities are calculated using Eq. 28, where 
the number of events is taken to be the sum of events over each of the sets, thus assuming each 
set to form a single bin. Now since the number of /x~ events in vacuum will be nearly equal to the 
number in matter with inverted hierarchy, a signal for matter effect is equivalent to a signal for the 
mass hierarchy when using only fj,~ events. This is brought out in Table 4, where ctnh-ih is seen to 
be close to ciNH-vac- Note that this property is specific to a charge-discriminating detector. Hence 
in the subsequent discussion, we describe the values of (7NH-vac only as a measure of sensitivity to 
matter effects as well as to the mass hierarchy, when working with only /i" events. 

For the sum of /i~ events in the 4 highlighted bins, a large value of (TNH-vac = 4.0(T is obtained, 
as seen in the second row of Table 4 for the standard value sin^ 2^13 = 0.1. For the 20 non- 
highlighted bins, the corresponding values of event numbers are Nyac = I860 and Nnh = 1831, 
giving a small UNH-vac = 0.7a. If the sum of events of all 24 bins is taken, N^ac = 2121 and 
Nnh = 2035, corresponding to a signal of o"NH-vac = l-Qa. Thus the 4 highlighted bins give a 4a 
signal for matter effect, which also indicates their sensitivity to the mass hierarchy. The above 
values are for sin^ 26*13 = 0.1. Table 4 also gives the values of NH- vacuum and NH-IH sensitivity 
for the 4 highlighted bins for two other values of ^13. 

In calculating the sensitivities shown in Tables 4 and 5, we have assumed that the neutrino 
mass-squared differences and mixing angles will be measured with good precision. Hence no 
marginalization over these parameters was done in calculating the sensitivities. The ability of a 
detector to rule out the wrong hierarchy hypothesis, however, does depend on the precision with 
which the neutrino parameters were measured at that point. Lower precision leads to reduced 
sensitivity. 

Summing over the four highlighted bins in Table 3, leads to integrating over large ranges in 
energy and in pathlength. Hence the detector resolution is not crucial in determining the number 
of events and the sensitivity. However, the sensitivity is likely to be better if the detector has 
good energy and pathlength resolution. The dependence of the sensitivity on detector resolution 
is currently under study [82]. 

Figure 15, for the same energy/baseline ranges and parameter values shows the event distri- 
butions for muons and antimuons versus L/E. 

For a charge-discriminating detector, the data for the number of /j,^ events would also be 
available separately. If the number of /x"*" events is taken into account and the quantity (N^- — N^+) 
is considered as an observable, then the sigma sensitivity is defined as follows: 



Using this, the NH-IH sensitivity is seen to improve in comparison to the sensitivity obtained using 
only /I" events, as seen by comparing Table 5 with Table 4. However, the NH-vacuum sensitivity 
suffers a decrease by this method. 

The above assumes that the cosmic ray fluxes will be well measured in ten years' time and the 
atmospheric neutrino fluxes can be predicted with much smaller errors than currently available. If 
the uncertainty in atmospheric neutrino flux prediction remains high, then one can use Up/Down 




(29) 
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sin^ 2^13 


N 

■■■'vac 


^^mat 


^^mat 


C'NH-vac 


cnh-ih 


0.05 


260 


227 


264 
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2.5(7 
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261 


204 


262 


4.0(7 


4.1(7 


0.2 


263 


163 


261 


7.8(7 


7.7(7 



Table 4: Number of n events in vacuum and in matter (Normal Hierarchy and Inverted Hierarchy) and corre- 
sponding values of a sensitivity computed by comparing NH-vacuum and NH-IH for 3 different values of sin^ 2(?i3 
for the 4 highlighted bins in the E and L range E = 5 - 10 GeV, L = 6000 - 9700 Km in Table 3. A31 = 0.002 eV^ 
and sin2 2^23 = 1-0. 
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104 
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Table 5: Number of ^~ and /z+ events in vacuum and in matter (Normal Hierarchy and Inverted Hierarchy) and 
corresponding values of NH-vacuum and NH-IH a sensitivity computed by considering the quantity (N^- — N^+) 
as an observable, for the same values of 613 and E and L ranges as in Table 4. A31 = 0.002 eV^ and sin^ 2^23 = 1-0. 
This depicts the gain in NH-IH sensitivity if the fi~ as well as /i+ events are taken into account and the difference 
is taken to be the observable. 
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Table 6: Number of /x~ events in vacuum and in matter (Normal Hierarchy), number of Down (unoscillated) fj,~ 
events and values of NH-vacuum a sensitivity computed using ratio of Up/Down events for the same values of ^13 
and E and L ranges as in Table 4. A31 = 0.002 eV^ and sin^ 26I23 = 1-0. 
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Figure 16: Ratio of atmospheric upward and downward going muon (/i ) events plotted against L for energy 
range of 5 GeV to 10 GeV and baselines between 6000 Km to 9700 Km. 



L = 8000 to 1 0700 Km, E = 4 to 8 GeV 
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Figure 17: Atmospheric muon events plotted against L for energy range of 4 GeV to 8 GeV and baselines between 
8000 Km to 10700 Km. The numbers in the legend correspond to the integrated number of muon events for the 
restricted range of L and E in matter and in vacuum for various values of sin^ 2^13 (in parenthesis). 



event ratios to cancel the normalization uncertainty of the flux predictions. Since the atmospheric 
neutrino fluxes depend only on the modulus of the cosine of the zenith angle, the muon event rates 
expected in case of no oscillations can be directly obtained from the experiment, by measuring 
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Figure 18; The total event rate for muons in matter and in vacuum plotted against Logio(L/E) for the restricted 
choice of L and E range (see Figure 17). The numbers in the legend correspond to the integrated number of muon 
events for the restricted range of L and E in matter and in vacuum for various values of sin^ 20i3 (in parenthesis). 



the rates of downward going muons, binned according to the same energy and the same value 
of I cos^|. Thus the downward going neutrinos provide the necessary information on unoscillated 
fluxes. Figure 16 shows the Up/Down muon event ratio versus L for L = 6000-9700 Km, E = 
5-10 GeV in vacuum and in matter for both signs of A31. For this range of energies and baselines, 
the downward event rates are calculated to be 410 for muons and 164 for anti-muons (same in 
matter and in vacuum). As mentioned earlier, the upward event rate for muons is 204 in matter 
and 261 in vacuum. For matter oscillations, this gives Up/Down = 0.50 ± 0.04, which differs 
from the corresponding ratio of 0.64 for vacuum oscillations by ~ 3.5cr. Table 6 lists the values 
of NH- vacuum sensitivity using the ratio of Up/Down events for the E and L ranges discussed 
earlier, for three different values of Oy^. 

As an example of how the muon survival rate can rise above the expected vacuum value, we 
show in Figure 17 the muon event rate for the baseline range of 8000-10700 Km and the energy 
range of 4-8 GeV. The solid curve is the vacuum event rate for sin^ 26'i3 = 0.1, while the three 
others are the muon survival rates incorporating matter effects for values of sin^ 2^13 = 0.05, 0.1 
and 0.2 respectively^^. The curves exhibit the rise above the vacuum survival rate visible in the 
energy range 4-7 GeV in the bottom panel of Figure 4(a). In addition, the curves demonstrate the 
sensitivity to ^13 which is exhibited in the probability plot given in Figure 6(b) in the energy range 
4-7 GeV. For values of sin^ 26^13 close to the present CHOOZ bound, a > 3cr signal for matter effect 
is visible in Figure 17. A similar ^13 sensitivity cum event rate plot is also given in Figure 18 with 
the muon rate now plotted versus L/E. Here it is seen that N^- is particularly sensitive to 6*13 in 



-'^^Note that the vacuum curve will not change appreciably with a change in ^13 , so it serves as the reference plot 
for vacuum. 
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Figure 19: Figure (a) and (b) depict the ratio of atmospheric muon and anti-muon events plotted against L for 
two different wide energy ranges (5 - 20 GeV and 3-10 GeV) and two basehne ranges (6000 - 9700 Km and 4000 
- 9000 km respectively). 
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Figure 20: Ratio of atmospheric muon and anti-muon events plotted against L for energy range of 4 GeV to 8 
GeV and baselines between 6000 Km to 10700 Km. For the restricted energy range considered here, the ratio for 
the case of matter oscillations with NH is greater than that of the vacuum ratio, when L is greater than 9000 Km. 
This is because of the increase of P^;^ caused by the sharp fall in P^jt-- 



the range 3.21 — 3.34 of Logio(L/E). The value for vacuum oscillations is 23 and it rises to 43 when 
matter effects are included, even for as small a value as sin^ 2^13 = 0.05. Treating this Logio(L/E) 
range as a single bin, o"NH-vac = 3.0 for sin^ 26^13 = 0.05, indicating a large signal for matter effect 
in this bin. Table 7 gives the values of o"NH-vac for this Logio(L/E) bin for three different values of 
6'i3, and Table 8 gives the corresponding values of the sensitivity using the Up/Down event ratios. 
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Table 7: Number of /i events in vacuum and in matter (Normal Hierarchy) and corresponding values of NH- 
vacuum a sensitivity for 3 different values of sin^ 29\3 for the Logio(L/E) range 3.21 - 3.34 shown in Figure 18 (E 
and L restricted between E = 4 - 8 GeV, L = 8000 - 10700 Km). A31 = 0.002 eV^ and sin^ 26*23 = 1-0. 
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Table 8: Number of /i events in vacuum and in matter (Normal Hierarchy), number of Down (unoscillated) /i 
events and values of NH-vacuum a sensitivity computed using ratio of Up/Down events for the same values of ^13 
and Logio(L/E) range as in Table 7. A31 = 0.002 eV^ and sin^ 26123 = 1-0. 



This effect may be looked for provided the L/E resolution is sufficient to observe it. We note that 
a significant rise of the muon survival rate in matter over its vacuum values is a signal of the size 
of the matter effect in P^,- overcoming that due to P^e? spite of the fact that the latter is close 
to resonance values (Figure 4(a), in the energy range 4-7 GeV). 

Further demonstration of the sensitivity to the sign of A31 and of the effects discussed in this 
paper can be gleaned from plots of the ratio of muon to anti-muon event rates [N^-/N^+] [43]. In 
Figure 19(a) we show this ratio for the baseline range 6000-9700 Km and the energy range 5-20 
GeV. The dominant effect here is due to the reduction in P™^ stemming from a large enhancement 
of P™ , (as evident in the probabihty plots for 7000 Km in Figure 4(b) above) and this is clearly 
visible in the event rates. Comparing the curves for positive and negative A31, we see that the 
peak to peak difference between them is about 90%. 

Selecting different (lower) ranges in both baselines and energy, we show the rate ratio for 4000 
- 9000 Km and 3-10 GeV in Figure 19(b). As evident from the discussion of the previous section, 
the P™^ effects develop after 8000 Km for the most part, hence this choice is primarily suited 
to demonstrate the effect of P™ on P™^. The influence of P™^ is, however, visible at the higher 
baselines, as it draws the A31 > curve close to the vacuum curve because, as discussed above, 
the P™ and P™ effects work in the opposite direction in this region. The dip-peak separation for 
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the curves for A31 > and A31 < is about 60% in this case. 

Finally, in an attempt to show an example of a baseline/energy range where both effects, i.e.the 
effect where P™^ plays a major role by raising the rate ratio above its vacuum value, and the effect 
where the resonant rise in PJ^ dominates over P™^ to bring about a decrease in N^- /N^+ compared 
to its value in vacuum, we choose the baseline range 6000-10700 Km and the energy range 4-8 
GeV (Figure 20). For A31 > 0, the dip in the ratio is maximum around 7000 Km and originates 
from the drop in the probability visible in the bottom panel of Figure 4(b), while the rise above 
the vacuum curve, which is maximum around 9700 Km, is a manifestation of the increase in 
P^^ consequent to the leirge decrease in P^^ occurring in the range 4-8 GeV, shown in 
Figure 4(a) and 6(b). 

4 Conclusions 

To conclude, we summarize the salient points of our study: 

• In addition to P^e, interesting and appreciable matter effects can arise in P^r at baselines 
> 7000 Km. These can synergistically combine to give correspondingly large effects in P™^. 
As a result, observably large signals may appear in a charge discriminating detector capable 
of measuring muon survival rates, providing a useful handle on the sign of A31. 

• The resonance amplified matter effects in P"!^ may be observable in future experimental 
facilities with intense beams and detectors capable of r identification. 

• We find that both P™^ and P™^ are sensitive to ^13. We have identified baselines and energies 
where the matter effects are amplified by the resonance and the sensitivity to ^13 is high. 
We have also studied the 623 dependence of P™^ and P™^. 

• We have performed a detailed study of parameter degeneracies at very long baselines, using 
numerical results with a realistic earth density profile. We find that for these baselines and 
at energies for which the criterion I.27A31L/E — mr/2 is satisfied, P™ is largely free of 
the degeneracies which could obscure measurements of ^13 and sign(A3i). The (^cp^^ia) 
degeneracy is lifted for P™^ and P™^ also. The sign(A3i) degeneracy vanishes for P™^ at such 
baselines and energies, but can be present in P™^ for some specific values of ^^13 and 6cp- 
We also note that the determination of ^13 from P™^ docs not suffer from the {623, tt/ 2 — 623) 
degeneracy, P™^ being a function of sin 2^23- Further, for the baselines and energies relevant 
for us this degeneracy disappears from P™^ and P™ as well. 

• In the second part of the paper we perform detailed calculations of muon survival rates for 
atmospheric neutrinos, using a 100 kT charge-discriminating iron calorimeter as prototype. 
Our choice here is guided by our motivation to explore whether it is possible to realistically 
determine the sign of A31 in an atmospheric neutrino experiment, over a timescale shorter 
than that anticipated for neutrino factories. 
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• While atmospheric neutrinos make available a wide range of baselines and energies, we find 
that in order to detect matter effects using total event rates a careful screening and selection 
of L and E is necessary to overcome the lack of a high intensity beam. We use our earlier 
discussion of matter effects in P™ , P™^ and P™^ as a guide to identify fairly broad L and E 
ranges where the sensitivity to the sign of A31 is significantly large. As an example, P™^ and 
P™ combine constructively in the range 5-10 GeV and 6000-9700 Km to yield a potentially 
large signal (~ Aa) for matter effect for A31 > (~ 3.5(7 for the Up/Down ratio). We 
also identify regions where an appreciable sensitivity to ^13 exists. For a large mass iron 
calorimeter type detector, a 3(7 discrimination is seen in the 3.2 — 3.35 range of Logio(L/E) 
for sin^ 2^13 as low as 0.05. 

• We have discussed the possibility of detection of matter effects and the sign of A31 using 
a charge discriminating iron calorimeter. Although these effects appear only for z/^ for A31 
positive (and only for for A31 negative), it may also be possible to look for them using high 
statistics mega-ton water Cerenkov detectors like Hyper-Kamiokande in Japan and UNO in 
the US [27, 34] . These effects may also be searched for in the accumulated SK data. 
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